Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/immich: init at 1.79.1 #244803

Closed
wants to merge 3 commits into from
Closed

nixos/immich: init at 1.79.1 #244803

wants to merge 3 commits into from

Conversation

oddlama
Copy link
Contributor

@oddlama oddlama commented Jul 22, 2023

Description of changes

This is an early draft to add a package and module for Immich, a very popular self-hosted photo and video backup solution. I would appreciate any feedback on module ergonomics. So far, all important settings are already exposed in the module and the packages should build correctly.

TODOs:

  • The nixos test would require network access and could benefit from persisting the machine learning models if that is possible (otherwise this is a multi-GB download each time the test is run)
  • Uploading images / videos could invoke external tools such as ffmpeg, exiftool, ..., and at this point I haven't tested if all required tools are propagated.
  • An update script is required to also update the sub-packages and their npmDepHashes.
  • aarch64 support should be possible and should be tested.
Things done

Early draft, not everything can be tested yet.

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 23.11 Release Notes (or backporting 23.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

@github-actions github-actions bot added 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: documentation 8.has: changelog 8.has: module (update) This PR changes an existing module in `nixos/` labels Jul 22, 2023
@oddlama oddlama added 8.has: package (new) This PR adds a new package 8.has: module (new) This PR adds a module in `nixos/` and removed 8.has: module (update) This PR changes an existing module in `nixos/` labels Jul 22, 2023
@github-actions github-actions bot added the 8.has: module (update) This PR changes an existing module in `nixos/` label Jul 22, 2023
@ofborg ofborg bot added 11.by: package-maintainer This PR was created by the maintainer of the package it changes 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 labels Jul 22, 2023
@oddlama oddlama force-pushed the init-immich branch 2 times, most recently from 613f47c to 4584e5f Compare July 23, 2023 11:31
@AndersonTorres
Copy link
Member

Split the commit

  • one for the package
  • one for nixos module
  • one for module tests

@oddlama
Copy link
Contributor Author

oddlama commented Jul 28, 2023

updated to 1.70.0, added automatic update script, split into separate commits

@oddlama oddlama changed the title nixos/immich: init at 1.68.1 nixos/immich: init at 1.70.0 Jul 28, 2023
@qbit
Copy link
Contributor

qbit commented Aug 7, 2023

It seems nodePackages.immich is a thing, not sure if the -cli bit should be changed to use that or not?

@oddlama
Copy link
Contributor Author

oddlama commented Aug 7, 2023

It seems nodePackages.immich is a thing, not sure if the -cli bit should be changed to use that or not?

That's the same thing that can be built from source here, just downloaded from the package registry. There currently are several cli implementations, and the one packaged here is the endorsed CLI that will replace all the others in the future.

Comment on lines +81 to +84
${optionalString serverCfg.typesense.enable ''
export TYPESENSE_API_KEY=$(cat ${serverCfg.typesense.apiKeyFile})
''}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

immich-server looks for TYPESENSE_API_KEY even if TYPESENSE_ENABLE=false, so when cfg.server.typesense.enable = false, immich-server won't start.

@mattmelling
Copy link
Contributor

I can get as far as the admin user sign up screen, hitting sign up causes an xhr to fail with a 500

@Atemu
Copy link
Member

Atemu commented Aug 23, 2023

This needs a rebase as it does not build with the newer versions of its dependencies we have in master right now.

@oddlama
Copy link
Contributor Author

oddlama commented Aug 23, 2023

Rebased and updated to 1.74.0. Beware that it doesn't build at this moment, because some transitive python dependencies are pulling in different versions of protobuf (which to my knowledge would require updates in nixpkgs before this will work again).

@oddlama oddlama changed the title nixos/immich: init at 1.70.0 nixos/immich: init at 1.74.0 Aug 25, 2023
@oddlama oddlama changed the title nixos/immich: init at 1.74.0 nixos/immich: init at 1.75.1 Aug 26, 2023
@Scrumplex
Copy link
Member

I'm slowly starting to question whether we should bother packaging at the speed at which upstream moves...

I am personally very interested in this, as I don't really like my current Docker-Compose and previously Helm deployments of Immich.

At least upstream changes mostly reduce complexity, although the Postgres extension does feel a bit more annoying to implement than having a separate daemon

@oddlama
Copy link
Contributor Author

oddlama commented Dec 9, 2023

Sorry for the slow update on this, I'm unfortunately quite busy at this time of year :/

I'm slowly starting to question whether we should bother packaging at the speed at which upstream moves...

I agree, this is becoming a concern more than I initially assumed. I've been in close contact with the immich devs for some time, so I've been aware of these big changes already, but they are still pushing out new stuff so insanely fast that it is hard to keep up.

Their architecture is just very much in motion right now and their development process (while very effective) is just docker-centric. That means there are quite some things in immich which assume exactly those conditions and that makes the module development a bit rough sometimes 😅. The current lack of E2E tests (which they are working on) means that I'll only discover these cases at runtime.

What concerns me most regarding this module is that each time they change something about their architecture (which is often) it will likely cause downstream breaking changes for users of this module.

I am personally very interested in this, as I don't really like my current Docker-Compose and previously Helm deployments of Immich.

Me too (obviously :P). But as long as native deployments are not a core goal of immich this will mean a lot of work and breakages on the NixOS side - and if I understand correctly that won't be the case for a long time, if ever.

Maybe the way to go forward here is to convert this draft to its own immich-nixos project that we can update more frequently and easily by ad-hoc packaging what is needed. That way we don't have to wait until all dependencies are available in NixOS proper. If anyone has their own thoughts on this matter, please share them :)

I'll definitely try to get all the changes in here sometime, but it'll take some work.

@christoph-heiss
Copy link
Contributor

Maybe the way to go forward here is to convert this draft to its own immich-nixos project that we can update more frequently and easily by ad-hoc packaging what is needed. That way we don't have to wait until all dependencies are available in NixOS proper. If anyone has their own thoughts on this matter, please share them :)

That definitely sounds like a idea to pursue. I'd also would very much like to deploy/use Immich and be ready to help with such an effort.

You already listed a bunch of good reason for this, and having the possibility to move fast (just like upstream) would make things a lot easier IMHO. In addition to that, it'd enable much easier collaboration on the effort to keep up with upstream and provide a working version.

If, in the future, the rapid development with breakages all the time slows down a bit, it can still be integrated in nixpkgs after all.

Just my 2ct.

@diogotcorreia
Copy link
Member

For those running Immich inside Docker but with PostgreSQL under NixOS, to update to 1.91.0 you need the pgvecto.rs PostgreSQL extension.

I've managed to (kinda) package it with the release binary (I did not try to build from source, seems to be a bit complicated because you need cargo-pgrx).
Here is the commit in my config updating to 1.91.0 if you want to copy the package: diogotcorreia/dotfiles@7676201
Suggestions are welcome!

@jvanbruegge
Copy link
Contributor

pgvecto.rs is now in nixpkgs: #281192
Would be nice to have immich as well

@wegank wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Mar 20, 2024
@bhankas
Copy link
Contributor

bhankas commented Mar 22, 2024

Since there seems to be an agreement on separate project being better suited for now, should we ask for a repo under nix-community? Ideally that would be better, but if we don't want that to be a blocker, we can always get started with any of the contributor's github and then transfer ownership. What say?

@Patryk27
Copy link
Member

should we ask for a repo under nix-community

I think having flake.nix inside the official repository would be more useful, if upstream is willing.

@bhankas
Copy link
Contributor

bhankas commented Mar 22, 2024

Since upstream devs don't use Nix for development (otherwise they'd already have a *.nix file), it would be undue burden on them to either keep the flake updated, or keep reviewing+accepting PRs for it, or provide someone full commit rights. I believe that is not a reasonable ask.

Besides, the idea behind separate repo for nixospackage+module is to be detached from both up streams (immich and nixpkgs) so we can keep working on the module itself.

Combined with f-droid allowing exact version of app to be and remain installed, both client and server can stay in sync until the package+module gets updated for upstream immich changes.

@adamcstephens
Copy link
Contributor

Since there seems to be an agreement on separate project being better suited for now, should we ask for a repo under nix-community? Ideally that would be better, but if we don't want that to be a blocker, we can always get started with any of the contributor's github and then transfer ownership. What say?

Just create a repo, start building and invite maintainers if you want. No need for it to start under nix-community.

@pinpox
Copy link
Member

pinpox commented Mar 23, 2024

Since there seems to be an agreement on separate project being better suited for now, should we ask for a repo under nix-community? Ideally that would be better, but if we don't want that to be a blocker, we can always get started with any of the contributor's github and then transfer ownership. What say?

Just create a repo, start building and invite maintainers if you want. No need for it to start under nix-community.

Feel free to ping or invite me, I'd like to test this.

@PowerUser64
Copy link
Contributor

An example of this being done in the past is when the cemu emulator was made open-source https://github.com/yboettcher/CemuFlake.

The upstream project was changing fast because of growing pains (running on linux for the first time), and there was a naming conflict with cemu-ti (formerly just called cemu) in our repo, so this flake was made in the meantime. Sounds like similar action would work well here too.

@GetPsyched
Copy link
Member

Have there been any updates on this effort?
cc @oddlama

@oddlama
Copy link
Contributor Author

oddlama commented Jun 4, 2024

No sorry, currently I don't have this updated. Immich has been moving extremely fast recently, and they often introduce new dependencies which (especially with python ML stuff) requires a lot of work to package. I haven't had the time to keep it up with their pace.

If anyone wants to pick this up, feel free to do so. But be aware that this won't be a walk in the park to keep updated :P
Maybe I'll revisit this when immich goes stable.

@anoadragon453
Copy link
Member

anoadragon453 commented Jun 16, 2024

Here is my NixOS Immich setup in case it is useful to anyone here. I am currently running Immich v1.106.4.

https://github.com/anoadragon453/dotfiles/blob/main/modules/server/immich.nix

Note that to get pgvecto-rs to build (a postgres extension that Immich relies on) using current nixos-unstable, #320302 is needed. This PR is now merged and built on nixos-unstable.

@RomeoV
Copy link
Contributor

RomeoV commented Jul 1, 2024

Thanks @anoadragon453, this looks like it works for me. For others struggling to get it to work:

  1. Copy Andrew's dotfile into e.g. /etc/nixos/immich.nix. If you typically use a stable release, change pkgs to pkgs_unstable and add a line
{config, lib, pkgs_unstable, ...}:
let
  pkgs = pkgs_unstable;
in 
...
  1. Add something like this to your configuration.nix
{ config, lib, pkgs, ... }: 
let
  unstable = import <nixos-unstable> {};
  pkgs_unstable = unstable.pkgs;
in {
  imports =   [
    (import /etc/nixos/immich.nix {inherit lib config pkgs_unstable; })
  ];

  services.immich = {
    enable = true;
    domain = "immich.your-domain.xyz";
    port = 8006;
    metricsPortServer = 8009;
    metricsPortMicroservices = 8010;
    storagePath = "/mnt/storage-box/immich";
    logLevel = "log";
  };
  1. et voilà

@jvanbruegge
Copy link
Contributor

jvanbruegge commented Jul 1, 2024

If someone does not want to run docker containers for immich and would like a native nixos package instead, I've packaged the current version (v1.106.4) and also built a nixos module.

If you use flakes, you should be able to include the module like this:

{
  description = "NixOS configuration";

  inputs = {
    nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable";
    immich.url = "github:jvanbruegge/server-config";
    immich.inputs.nixpkgs.follows = "nixpkgs";
  };

  outputs = inputs@{ nixpkgs, immich, ... }: {
    nixosConfigurations = {
      hostname = nixpkgs.lib.nixosSystem {
        system = "x86_64-linux";
        modules = [
          ./configuration.nix
          immich.nixosModules.immich
        ];
      };
    };
  };
};

You can then use it like this:

services.immich = {
  enable = true;
  host = "0.0.0.0";
  port = 8080;
  mediaLocation = "/mnt/storage-box/immich";
  openFirewall = true;
  secretsFile = "/run/secrets/immich"; # This has at least DB_PASSWORD=<pass>
};
sops.secrets.immich = {};

The flake also exports just the package (without the module). I have this running for a few weeks now on my own server.

@Atemu
Copy link
Member

Atemu commented Jul 1, 2024

@jvanbruegge Would you be willing to create a PR for the package and optionally the module too?

@marcusramberg
Copy link
Contributor

The flake also exports just the package (without the module). I have this running for a few weeks now on my own server.

https://github.com/jvanbruegge/server-config/blob/5ee96717d8476c6671bb1c28d144007bdf74915b/flake.nix#L41 seems like your flake currently does not export the module at all?

@jvanbruegge
Copy link
Contributor

seems like your flake currently does not export the module at all?

Sorry about that, that got lost in rebasing. Fixed now.

Would you be willing to create a PR for the package and optionally the module too?

Sure, I can do so soon

@Pablito2020
Copy link
Contributor

seems like your flake currently does not export the module at all?

Sorry about that, that got lost in rebasing. Fixed now.

Would you be willing to create a PR for the package and optionally the module too?

Sure, I can do so soon

You need to remove the "self" attribute, just like I did in this commit.

I also noticed that you use your own DB instead of the one declared in your module. This works on your machine, but the default password in the module DB is empty, which Immich doesn't allow. I added a default password in this commit. This is a quick fix; we can later get it from the module input.

You can test it with the following configuration:

  services.immich = {
    enable = true;
    host = "0.0.0.0";
    port = 8080;
    mediaLocation = "/data/immich";
    openFirewall = true;
    environment = {
      DB_USERNAME="immich";
      DB_PASSWORD="password123";
    };
    database.createDB = true;
    secretsFile = config.age.secrets.immichDatabase.path; # Right now this file is empty, since we pass the DB_PASSWORD in environment... Dirty hack as I said.
  };

Additionally, we need to fix the permissions of the mediaLocation folder. I had to run:

$ chmod -R 0777 /data

And voilà! Immich is working on my machine.

Thanks a lot for the module! Coming from your dotfiles, I only needed a bit of tweaking to get it working. Making it an official module won't take much time, though maintaining it might be challenging since Immich evolves quickly and with breaking changes.

Cheers.

@christoph-heiss
Copy link
Contributor

@Pablito2020

$ chmod -R 0777 /data

Please never do that! Are you aware of the consequences?
It should be owned by the immich user & group only, without allowing every users accessing it.

@Pablito2020
Copy link
Contributor

@Pablito2020

$ chmod -R 0777 /data

Please never do that! Are you aware of the consequences? It should be owned by the immich user & group only, without allowing every users accessing it.

Yes, I am aware of the consequences of giving rwx access to all users of the machine, it was just for getting it to work, someone should add the privileges to the systemd-service inside the module.

I was just trying to tell what's missing inside the module for making it work on the first try.

I am sorry if my comment wasn't clear.

@christoph-heiss
Copy link
Contributor

Derailed into OT now completely, but:

Yes, I am aware of the consequences of giving rwx access to all users of the machine, it was just for getting it to work, someone should add the privileges to the systemd-service inside the module.

It is just that people really like to blindly run random commands people post on the internet without knowing what they do, inevitable resulting in losing data or such in the end ..

Sometimes you (unfortunately!) have to protect people from themselves.

@jvanbruegge
Copy link
Contributor

jvanbruegge commented Jul 2, 2024

I've now opened the PR for the package and module here: #324127

Differences to the version in my repo:

  • Updated the version to v1.107.1 released a few minutes ago
  • The cli executable is just called immich as that is what the immich docs use
  • All configuration options in the module have documentation now
  • The postgres config by default uses the unix socket now, meaning that services.immich.enable = true is enough to get a working service.
  • I've added a nixos test that creates an admin user, creates an API key for that admin user and then uses the immich-cli to get the server-info (so that both server and cli get tested)

@wegank wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.status: merge conflict This PR has merge conflicts with the target branch 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 8.has: changelog 8.has: documentation 8.has: module (new) This PR adds a module in `nixos/` 8.has: module (update) This PR changes an existing module in `nixos/` 8.has: package (new) This PR adds a new package 10.rebuild-darwin: 1-10 10.rebuild-linux: 1-10 11.by: package-maintainer This PR was created by the maintainer of the package it changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.